Evaluating spoken dialogue agents with PARADISE: Two case studies
نویسندگان
چکیده
This paper presents PARADISE PARAdigm for DIalogue Sys tem Evaluation a general framework for evaluating and comparing the performance of spoken dialogue agents The framework decou ples task requirements from an agent s dialogue behaviors supports comparisons among dialogue strategies enables the calculation of per formance over subdialogues and whole dialogues speci es the relative contribution of various factors to performance and makes it possible to compare agents performing di erent tasks by normalizing for task complexity After presenting PARADISE we illustrate its application to two di erent spoken dialogue agents We show how to derive a per formance function for each agent and how to generalize results across agents We then show that once such a performance function has been derived that it can be used both for making predictions about future versions of an agent and as feedback to the agent so that the agent can learn to optimize its behavior based on its experiences with users over time
منابع مشابه
PARADISE: A Framework for Evaluating Spoken Dialogue Agents
This paper presents PARADISE (PARAdigm for Dialogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to perf...
متن کاملEvaluating Spoken Language Systems
Spoken language systems (SLSs) for accessing information sources or services through the telephone network and the Internet are currently being trialed and deployed for a variety of tasks. Evaluating the usability of different interface designs requires a method for comparing performance of different versions of the SLS. Recently, Walker et al (1997) proposed PARADISE (PARAdigm for DIalogue Sys...
متن کاملParameters for Quantifying the Interaction with Spoken Dialogue Telephone Services
When humans interact with spoken dialogue systems, parameters can be logged which quantify the flow of the interaction, the behavior of the user and the system, and the performance of individual system modules during the interaction. Although such parameters are not directly linked to the quality perceived by the user, they provide useful information for system development, optimization, and ma...
متن کاملThe Utility of Elapsed Time as a Usability Metric for Spoken Dialogue Systems
It is commonly assumed that elapsed time is an important objective metric for evaluating the performance of spoken dialogue systems. However, our studies based on the PARADISE framework consistently find that other predictors are stronger contributors to user satisfaction than elapsed time. In this paper, we show that several possible explanations for this apparently counter-intuitive finding a...
متن کاملEvaluating Dialogue Strategies in a Spoken Dialogue System for Email
This paper presents an evaluation of directed dialogue (DD) and mixed initiative (MI) strategies in a spoken language system for Email. We compare the DD strategy, in which the system controls the dialog, to the MI strategy, in which users can flexibly control the dialog. For evaluating both strategies we used the PARADISE framework, which supports comparisons among dialogue strategies. Our exp...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Computer Speech & Language
دوره 12 شماره
صفحات -
تاریخ انتشار 1998